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Abstract. 

We consider families of cyclic covers of P 1 , where we fix the covering 
group and the local monodromies and we vary the branch points. We 
prove that there are precisely twenty such families that give rise to a 
special subvariety in the moduli space of abelian varieties. Our proof 
uses techniques in mixed characteristics due to Dwork and Ogus. 
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Introduction 

According to a conjecture of Coleman, if we fix a genus 5^4 and consider 
smooth projective curves C over C of genus g, there should be only finitely 
many such curves, up to isomorphism, such that Jac(C) is an abelian variety 
of CM type. This conjecture is known to be false for g G {4, 5, 6, 7}, by virtue 
of the fact that for these genera there exist special subvarieties S C srf g (also 
known as subvarieties of Hodge type) of positive dimension that are contained 
in the (closed) Torelli locus and that meet the open Torelli locus. 
All known examples of such special subvarieties S arise from families of cyclic 
covers of P 1 . As input for this we fix integers to > 2 and N ^ 4, together 
with an TV-tuple a = (ai, . . . , ajv); then we consider cyclic covers C t — > P 1 
with covering group Z/mZ, branch points ti,...,tjy in P 1 , and with local 
monodromy dj about ij. Varying the branch points we obtain an (N — 3)- 
dimensional closed subvariety Z — Z(m,N,a) C srf g given by the Jacobians of 
the curves C t , and for certain choices of (to, A, a) it can be shown that Z is a 
special subvariety. 

The main purpose of this paper is to give a list of monodromy data (to, N, a) 
such that Z(m,N,a) is special, and to prove that there are no further such 
examples. 

By construction, the Jacobians J t come equipped with an action of the group 
ring Z[/u m ]. This action defines a special subvariety S(/j, m ) C srf g that con- 
tains Z and whose dimension can be easily calculated in terms of the given 
monodromy data (to, N, a). We always have N — 3 dim S(fi m ), and if equal- 
ity holds Z = 5(yU m ) is special. To search for triples (to, N, a) for which 
dim S(/j, m ) = N — 3 is something that can be done on a computer, and it is 
therefore surprising that this appears not to have been done until recently. 
Thus, while examples in genera 4 and 6 have been given by de Jong and Noot 
in [6] — and in fact, these examples at least go back to Shimura's paper [24]— 
examples with g = 5 and g = 7 were found only much later; see Rohde [22]. At 
any rate, with the help of a computer program we find twenty examples with 
dim5(^ m ) = N — 3. The main result of this paper is that this list is complete: 

Main theorem. - Consider monodromy data (to, N, a) as above. Then the 
closed subvariety Z{m, N, a) C J^ g ,c is special if and only if (to, N, a) is equiv- 
alent to one of the twenty triples listed in Table 1 . 

Note that if N — 3 < dim S([i m ), which means that the inclusion Z C 5(/i m ) 
is strict, Z could a priori still be a special subvariety. The main point of our 
result is that even in these cases we are able to prove that Z is not special. 
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To get a feeling for the difficulty of the problem, consider, as an example, the 
family of (smooth projective) curves of genus 8 given by y 10 = x(x — l)(x — t) 2 , 
where t e T = Pj, \ {0, 1, oo} is a parameter. The corresponding family of 
Jacobians J — > T decomposes, up to isogeny, as a product J old x J ncw of 
two abelian fourfolds. (The old part comes from the quotient family u 5 = 
x(x — l)(x — t) 2 ; the new part is the family of Pryms.) Both J old and J ncw 
give rise to a 1-dimensional special subvariety in ^4, and they both admit an 
action by Z[£s] = Z<[CioL with the same multiplicities on the tangent spaces at 
the origin. A priori it might be true that J old and J new are isogenous, in which 
case the family J — > T would define a special subvariety in srf%. Our theorem 
says, in this particular example, that this does not happen. 
From the perspective of special points and the Andre-Oort conjecture, one 
could state the problem, in the example at hand, as follows. There are infinitely 
many values for t such that J t old is of CM type; likewise for J t new . Do there 
exist infinitely many t such that J° ld and J t new are simultaneously of CM type? 
Our theorem, combined with a result of Yafaev [25], implies the answer is no, 
at least if we assume the Generalized Riemann Hypothesis. See Corollary 3.7. 

Our proof of the main theorem uses techniques from characteristic p and is 
based on a method that in some particular cases has already been used by dc 
Jong and Noot in [6], Section 5. For a suitable choice of a prime number p, 
the Jacobians in our family will, at least generically, have ordinary reduction 
in characteristic p. Moreover, if we assume Z(m, N, a) C sf g to be special then 
by a result of Noot [18] we can arrange things in such a way that the canonical 
liftings of these ordinary reductions are again Jacobians. Already over the Witt 
vectors of length 2 this is a very restrictive condition that, using the results of 
Dwork and Ogus in [8], can be turned into something computable. 
In individual cases (such as the example sketched above, or the examples 
treated in [6]), these techniques give a rather effective method to prove that 
some given family of curves does not give rise to a special subvariety in £^ g .c- 
To do this for arbitrary data (to, N, a) is much harder, and some perseverance 
is required to deal with the combinatorics that is involved. It would be inter- 
esting to have a purely Hodge-thcorctic proof of the main theorem; as it is, 
we do not even have a Hodge-theoretic method that works well in individual 
examples. 

Let us give an overview the contents of the individual sections. In Section 1 
we quickly review the notion of a special subvariety and we summarize some 
facts we need. In Section 2 we introduce the families of curves that we want 
to consider. In Section 3 we discuss the special subvariety S(/j, m ) that contains 
Z(m,N,a), we give the list of data (m,N,a) for which Z(m,N,a) = S{^i m ) 
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and we state the main results. Sections 4 and 5 contain the main technical 
tools for the proof. In Section 4 we discuss how the VHS associated with our 
family of curves decomposes and we give some results about the monodromy of 
the summands. In Section 5 we briefly review the techniques of [8] and we give 
some refinements that we need in order to deal with hyperelliptic families. In 
Sections 6 and 7, finally, we prove the main theorem, first in the case N = 4, 
then for N ^ 5. We refer to the beginning of Section 6 for a brief explanation 
of how the proof works. 

Acknowledgements. I thank Frans Oort for his encouragement and for stimu- 
lating discussions. It is a great pleasure to dedicate this paper to him. I thank 
Dion Gijswijt and Fokko van de Bult for helpful suggestions related to the 
combinatorics involved in the proof of Lemma 7.2. Further I thank Maarten 
Hoeve, who wrote two Python programs (available on the author's webpage) 
to test some of the properties we need. The results of one of these programs 
are used in the proof of Lemma 7.2. Finally, I thank the referee for his or her 
comments on the paper and for some very helpful suggestions regarding the 
exposition. 

Notation. If x is a real number, (x) = x — \x\ denotes its fractional part. For 
a e Z/toZ or a 6 Z we denote by [a) m the unique representative of the class a 
(resp. the class a mod m) in {0, . . . , m — 1}. 

1. Some preliminaries about special subvarieties 

In this section we work over C. We recall the notion of a special subvariety. For 
further details we refer to [12], where the terminology "subvarieties of Hodge 
type" was used, and [20]. 

(1.1) Consider the moduli space trf g \ n \ OI g-dimensional principally polarized 
abelian varieties with a level n structure, for some n ^ 3. We first briefly recall 
the description of this moduli space as a Shimura variety. 
Let Vz := I? 9 C V := Q 2s , and let V z x V z -> Z be the standard symplectic 
form. Let G :— CSp^,^) be the group of symplectic similitudes. Let S := 
Resc/RG m be the Deligne torus, and let Sj be the space of homomorphisms 
h: S — > Gr that define a Hodge structure of type (—1,0) + (0, —1) on Vz for 
which ±(27ri) • \& is a polarization. The pair (Gq,$j) is a Shimura datum, 
and srfg can be described as the associated Shimura variety. Concretely, if 
K n := {g e G{±) \g = l mod n) then ^ SiW (C) £* G(Q)\fi x G(A fin )/^ n . 
In order to define special subvarieties, consider an algebraic subgroup H C Gq 
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such that 

Yh ■= {h £ Sj I h factors through #r} 

is non-empty. The group H(M) acts on Yh by conjugation. It can be shown 
(see [11], Section 1.3, or [12], 2.4) that Yh is a finite union of orbits under 
H(M). We remark that the condition that Yh is non-empty imposes strong 
restrictions on H ; it implies, for instance, that H is reductive. If Y + C Yh is 
a connected component and r]K n £ G(Aa n )/K n , the image of Y + x {r)K n } in 
80 g \n\ is an algebraic subvariety. 

(1.2) Definition. — A closed irreducible algebraic subvariety 5 C &? g ,[ n \ is 
called a special subvariety if there exist Y + C Yh and ?7if n e G(Aa n )/K n as 
above such that 5(C) is the image of Y + x {r/K n } in ^^(C). 

We refer to [12] and [20] for alternative descriptions and basic properties of 
special subvarieties. 

(1.3) As level structures play no role in what we are doing, we prefer to state 
our results in terms of the moduli stack srf g . Of course this is a stack, not 
a variety, but we allow ourselves to abuse terminology and speak of special 
subvarieties in srf g (over C), where "special substack" would perhaps be more 
correct. By definition, then, a special subvariety 5 C srf g is a closed, reduced 
and irreducible algebraic substack such that for some (equivalently: any) n ^ 3 
the irreducible components of the inverse image of 5 under the natural map 
8# g \n\ s$ g are special. (It is in fact enough to require that some irreducible 
component is special.) 

(1.4) Let /: X — > T be a family of g-dimensional abelian varieties over an 
irreducible non-singular complex algebraic variety T. Choose a principal po- 
larization A of X/T, and let <fr: T — > srf g be the resulting morphism. Whether 
or not the closure of 4>{T) is a special subvariety of srf g only depends on the 
isogeny class of the generic fibre of X/T as an abelian variety over the function 
field of T. If we replace X/T by an isogenous family or choose a different A, 
the morphism (j> is replaced by some other morphism </>': T — > srf g , but if 4>{T) 
is special then so is 4>'{T). (This is even true without the assumption that A 
is a principal polarization, but we shall not need this.) Without ambiguity we 
may therefore say that X/T is special if <fi(T) C srf g is special for some choice 
of a polarization. 

One property we shall use is that if X/T is isogenous to Y\ x F 2 and X is special, 
the factors Y t are both special. The converse is not true: if Y\ /T and Y 2 /T are 
special, it is not necessarily the case that (Yi x Y 2 )/T is special. 
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(1.5) Let f: X ->T be as in 1.4. Consider the Q-VHS over T with fibres the 
first cohomology groups H 1 (X t , Q). Let 6 G T(C) be a Hodge-generic point for 
this VHS, and let M C GL(i? 1 (X fc , Q)) be the generic Mumford-Tate group 
of the family. Choose a principal polarization A of X/T, let <j>: T — > be the 
resulting morphism, and write Z :— </>(T). Finally, let S C g/ g be the smallest 
special subvariety containing Z. 

In order to calculate the dimension of S, it suffices to know the adjoint real 
group M| d . More precisely, if M| d = Qi x • • • x Q r is the decomposition of 
this group as a product of simple factors, dim(S') = Yll=i d(Qi), where d(Qi) 
is a contribution that only depends on the isomorphism class of the simple 
group Qi. The only cases that are relevant for us are that 

• d(Q) — if Q is anisotropic (compact); 

• d(Q)=pqiiQ = PSU{p,q); 

• d(Q) = h(h + l)/2tfQ = PSp 2h . 

2. The setup 

In this section, given data (m, N, a) as in the introduction (see 2.1 below), we 
construct a family of cyclic covers of P 1 over some base scheme T. For later 
purposes we shall do this over a ring R of finite type over Z. 

(2.1) Let to and N be integers with to > 2 and N ^ 2, and consider an TV- 
tuple of positive integers a — (a\, . . . , ajv) such that gcd(m, a\, . . . , ajv) = 1- 
We further require that a, ^ modulo m for all i and ~Y^ =1 a% = modulo m. 
The triple (m, AT, a) serves as input for our constructions. 

We call two such triples (m,N,a) and (m',N',a') equivalent if m = ml and 
N = N' and if the classes of a and a' in (Z/ml*) N are in the same orbit under 
(Z/roZ)* x Sat. Here we let (Z/mZ)* act diagonally by multiplication, and 
the symmetric group 6 at acts by permutation of the indices. 
In what follows we shall usually assume TV > 4 but for some arguments it is 
useful to allow iV to be 2 or 3. 

(2.2) Let (m, N, a) be a triple as in 2.1. Let i? be the ring Z[l/m, w]/$ m with 
$ m the mth cyclotomic polynomial. We write ( E R for the class of u; it is 
a root of unity of order m. We embed R into C by sending £ to exp(27rz/m). 
The element ( defines an isomorphism of i?.-group schemes (Z/mZ)j? — > Mm.fl 
by (b mod to) i-> £ b . 

Let ?7 C (Ajj)^ be the complement of the big diagonals. In other words, U 
is the i?-schemc of ordered iV-tuples (ti, . . . ,£at) of distinct points in A 1 . Let 
B C Pfj be the projective curve over U obtained as the Zariski closure of the 
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affine curve whose fibre over a point (£1, . . . , £jv) is given by 

N 

y m = (x- h)^ ...(*- t N ) a " =]J(x ur ■ 

t=l 

We have a ^ m -action on £? over U by ( ■ (x,y) — (x, £ ■ y). The rational 
function x defines a morphism ttb- B — > P~. 

There exist an open subscheme T C f, a smooth proper curve /: C — >• T 
equipped with an action of /i m ,T, and a /z mj T-equivariant morphism p: C — > St, 
such that for every point £ £ T the morphism on fibres fa: Ct — > St is a 
normalization of £?i. Let 7r := 7Tb o p: C — > P T , which is a finite morphism that 
realizes P T as the quotient of C by the action of p m ,T- If the context requires 
it, we include the data (m,N,a) in the notation, writing C — C(m 7 N,a) for 
instance. Further we write J — > T for the Jacobian of C over T. 
If k is a field and £ = (£1, . . . , £jy) G we can also describe 7r t : C t — > Pj, as 

the ^i m -cover of P^ with branch points £1, . . . ,£jv and local monodromy about £j 
given by the element ( ai G /U m . 

The assumption that gcd(m, ai, . . . , oat) = 1 implies that the fibres of /: C — > T 
are geometrically irreducible. Let r, := gcd(m, ai). The Hurwitz formula gives 
that the fibres have genus 

(,.,.!, ^i + (A '- 2) "7^- r - . 

so we obtain a morphism ?/>: T — > ^ 3 over i?. Define 
(2.2.2) (p-.T-^s/g 

to be the composition of ^ with the Torelli morphism. Up to isomorphism, the 
morphism and hence also <fi, only depends on the equivalence class of the 
triple (m, N, a) for the equivalence relation defined in 2.1. 

(2.3) Remark. — For computational purposes we have restricted our attention 
to the case where all branch points £j are in A 1 C P , and we have not fixed 
any of these branch points, thereby creating some redundancy and excluding 
families where one of the branch points is fixed to be the point at oo. For our 
main results, the only thing that really matters is the closure of the image of 
(j)£\ Tc — > =2^,c- For instance, while the family of curves y 10 = x(x — l)(x — A) 
is not among the families we consider (the point at oo being a branch point), 
the subvariety of s^q we obtain from this family is the same as the one obtained 
by taking m = 10, with N — 4 and a = (1, 1, 1, 7). 
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(2.4) Convention. - In what follows we shall in several steps replace the 
base scheme T by a subschema. In such a case, it will be understood that 
we replace C by its restriction to the new base scheme, and we again write 
f:C—>T for the curve thus obtained. Similarly, we retain the notation for 
various other objects associated with our family of curves. 

(2.5) Notation. — Let M be a module over some commutative i?-algebra, or 
a sheaf on some i?-scheme, on which the group scheme \i m acts. For n £ Z/mZ 
we write 

M (n) := {xeM\ C(x) = C ■ x} , 

which in the sheaf case has to be interpreted on the level of local sections. We 
refer to M(„) as the n-eigenspace of M. We have M — (B n ez/mZ M( n y 

(2.6) Recall that = gcd(m, aj). Consider the /i m -cover ir: Ct — > P 1 for some 
t e T(k), where k is a field. For i G {1, . . . , N} and n G Z, let 



l(i,n) := -1 + 



Ti - nai 




—nai 


m 




m 



where the second equality easily follows from the fact that r» = gcd(m, a{). 
Consider the differential forms 

N 

(2.6.1) w n ,„ := y n -(x- hf ■ H(x - t i ) ,(i,n) • dx , 

i=l 

and note that these only depend on the pair (n mod m, v). 
The following result is standard. 

(2.7) Lemma. - Let n G Z/mZ loitft n / 0. TTie forms LJ n>v for ^ 
f < — 2 + J2i=i ( ) are regular l-forms on C t and they form a k-basis 
for H°{C u ^) (n) . 

3. The special subvariety given by the action of the covering group 

In this section we work over C and we fix a triple (m, N, a) as in 2.1 with N > 4. 
We retain the notation introduced in the previous section, with the convention 
that all objects are now considered over C via the chosen embedding R M> C. 

(3.1) Definition. — We let Z(m 7 N,a) C £/ g ,c be the scheme-theoretic (or 
rather, stack-theoretic) image of the morphism (f>: T — > &4 g of (2.2.2). 

In other words, Z(m, N, a) is the reduced closed substack of s^ 9l c with under- 
lying topological space <j>(T). We note that Z(m,N,a) only depends on the 
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equivalence class of (m, N, a) and does not depend on the choice of the open 
subscheme T in 2.1. The dimension of Z(m,N 7 a) equals N — 3. 

(3.2) Notation. — It will be convenient to write 

(3.2.1) I(m) := [(Z/mZ) \ {0 mod m}] /{±1} . 

Recall that (x) denotes the fractional part of x. For n G Z/mZ we define 

(3.2.2) ,„ := (-l + Ef =1 (^) if^O, 

lO ifn = 0, 

which by Lemma 2.7 is the dimension of the (n)-eigenspace of H°(C t , ft 1 ), for 
any t G T(C). 

(3.3) The substack Z(m,N,a) C ^ is contained in a special subvariety 
S{Hm) C ^ determined by the action of Z[^ m ] on the relative Jacobian 
J — >• T. More precisely, S{fx m ) is the largest closed, reduced and irre- 
ducible substack 5 G s4 g containing Z(m, N, a) such that the homomorphism 
Z[/i m ] — > End( J/T) induced by the action of \i m on C/T extends to an action 
of Z[/i m ] on the universal abelian scheme over 5. 

Choose a base point b G T'(C), and let (J&, A) be the corresponding Jacobian 
with its principal polarization. With (Vz,^) as in 1.1, choose a symplectic 
similitude a: Hi(Jb,Z) Vz, where we equip £fi(J(,,Z) with its Riemann 
form (i.e., the polarization, in the sense of Hodge theory, that corresponds 
with A). Via a, the action of fi m on Jf, induces a structure of a Q[/x m ]-module 
on V — Vz ® Q. Consider the algebraic subgroup H C Gq = CSp(F, \I/) of 
|-linear symplectic similitudes, i.e., the subgroup given by 



H -GLQ^j^nCSpCV,*). 

With notation as in 1.1, the image of Yh C $j under the map 

Sj G(Z)\Sj s G(Q)\i5 x G(A fin )/G(Z) = ^,(C) 

is (the set of C-points of) a finite union of algebraic subvarieties of £? g , and 
the special subvariety S(fi m ) is the unique irreducible component of this image 
that contains Z(m,N,a). 
The dimension of S{ji m ) is given by 



(3.3.1) dimS(^„) = + { 



if m = 2k is even, 



2 



if m is odd, 
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where the first sum runs over the pairs ±n G I(m) with 2n ^ 0. Sec [14], 

Section 5; see also Remark 4.6 below. 

In particular, as Z(m 7 N,a) C S(/j m ) we have 

(3.3.2) N -3 sCdim 5(/x m ) . 

If equality holds then Z(m,N,a) — S(/i m ) C «e/ g is a special subvariety; it 
then follows that among the Jacobians J tl for t G ^(C), there are, up to 
isomorphism, infinitely many Jacobians of CM type. (Here we use that on a 
special subvariety the CM points lie dense.) 

(3.4) Inventory of examples. — Table 1 lists twenty triples (m 7 N,a) with 
N ^ 4 for which N — 3 = dim5(^ m ), so that Z(m 7 N,a) C £? g ,c is a special 
subvariety. Our assumption that N ^ 4 means we are only considering the 
cases that give rise to a special subvariety of positive dimension. 

The first column of the table gives a number that we assign to each example 
for reference. The examples are sorted first by genus, then by degree of the 
cover. The second column gives the genus of the curves in the family. In the 
next three columns we give the data (m, N, a). In most cases there is a unique 
a = (di, . . . , oat) in its equivalence class such that l<ai<---<ajv<m — 1 
with J2 a i — 2m, and if there is a unique such representative, this is the one we 
list. If N — 4 there are usually two such representatives, the second one being 
(m — CL4, m — 03, m — 02, m — ai); we list the one which lexicographically comes 
first. In the last column we give references to places in the literature where the 
example can be found. Here "S(i)" refers to example (i) in [24], "Mi" refers 
to example i in the appendix of [17], and dJN(1.3i) refers to example (1.3i) 
in [6]. These examples can also be found in [22]; see 4.8 below. 

(3.5) Remark. — If we have a triple (m,N,a) as in 2.1 with N = 3, the 
associated subvariety Z = Z(m, N,a) is a special point in srf g . 

The following theorem is the main result of this paper. It says that the list of 
examples in Table 1 is exhaustive. 

(3.6) Theorem. - Consider data (m,N,a) as in 2.1, with N ^ 4. Then 
Z(m,N,a) C £^ g .c is a special subvariety if and only if (m,N,a) is equivalent 
to one of the twenty examples listed in Table 1. 

By what was explained above, it only remains to be shown that Z(m, N, a) is 
not special if (m, N, a) is not equivalent to one of the triples in the table. In 4.9 
we shall first prove this for m = 2. In Section 6 we shall prove the theorem for 
N = 4. The case N > 4 is treated in Section 7. 
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genus 


to 


N 


a 


references 


(l) 


1 


2 


4 


(1,1,1,1) 




(2) 


2 


2 


6 


(1,1,1,1,1,1) 




(3) 


2 


3 


4 


(1,1,2,2) 




(4) 


2 


4 


4 


(1,2,2,3) 




(5) 


2 


6 


4 


(2,3,3,4) 




(6) 


3 


3 


5 


(1,1,1,1,2) 


S(l), M41 


(7) 


3 


4 


4 


(1,1,1,1) 




(8) 


3 


4 


5 


(1,1,2,2,2) 


S(3), M43 


(9) 


3 


6 


4 


(1,3,4,4) 




(10) 


4 


3 


6 


(1,1,1,1,1,1) 


S(2), M23, dJN(1.3.1) 


(11) 


4 


5 


4 


(1,3,3,3) 


S(4), dJN(1.3.2) 


(12) 


4 


6 


4 


(1,1,1,3) 




(13) 


4 


6 


4 


(1,1,2,2) 




(14) 


4 


6 


5 


(2,2,2,3,3) 




(15) 


5 


8 


4 


(2,4,5,5) 




(16) 


6 


5 


5 


(2,2,2,2,2) 


S(5), M44 


(17) 


6 


7 


4 


(2,4,4,4) 


S(6), dJN(1.3.3) 


(18) 


6 


10 


4 


(3,5,6,6) 




(19) 


7 


9 


4 


(3,5,5,5) 




(20) 


7 


12 


4 


(4,6,7, 7) 





Table 1: monodromy data that give rise to a special subvariety 



Combining the theorem with the main result of [25] we obtain, for TV — 4, the 
following finiteness result for the number of CM fibres. 



(3.7) Corollary. - Assume the Generalized Riemann Hypothesis for CM 
fields. If we have a triple (to, N, a) with N = 4 and the equivalence class 
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of this triple is not among the twenty examples listed in Table 1 then up to iso- 
morphism there are finitely many Jacobians of CM type in the family J — > Tq. 

4. Decomposition of the variation of Hodge structure 

In this section we again work over C. We fix a triple (m,N,a) as in 2.1 with 
N ^ 4. The corresponding family of curves C — > T gives rise to a variation of 
Hodge structure over T (over C) on which Q[^i m ] acts. We describe the resulting 
decomposition of the VHS, and we give some results about the monodromy of 
the summands. 

(4.1) Let (m,iV, a) be as in 2.1. For n e Z/mZ, let N(n) be the number of 
indices i E {1, . . . , N} such that nai ^ modulo to. In particular, N(n) = N 
if n e (Z/mZ)*. If n 7^ then d n + cL„ = N(n) - 2. 

Let to' ^ 2 be a divisor of to, say with to = rm! . Write N' := N(r mod to), 
and consider the TV'-tuple a' in (Z/to'Z)^ obtained from the TV-tuple (ai mod 
to', . . . , on mod to') by omitting the zero entries. Then (to', N', a') is again a 
triple as in 2.1; we refer to it as the triple obtained from (to, N, a) by reduction 
modulo to'. 

(4.2) As in 2.2, consider the family of curves f:C^T associated with the 
triple (m,N,a), where, as in the previous section, we work over C. The coho- 
mology groups i? 1 (C t , Q) are the fibres of a polarized Q-VHS with underlying 
local system V :— R^f^Qc- This Q-VHS comes equipped with an action of 
the group ring Q[fi m ] = Y\d[ m K ^ where K d = Q[t]/$ d is the cyclotomic field 
of dth roots of unity. Accordingly we have a decomposition of Q-VHS, 

(4.2.1) V = ® dlm Y^ , 

where is a polarized Q-VHS over T equipped with an action of K d - 

(4.3) Let (to, N, a) be as in 2.1, with N ^ 4. Let to' ^ 2 be a proper divisor 
of to, and let (m',N',a') be the triple obtained from (m,N,a) by reduction 
modulo to', as in 4.1. 

Let p: A N — > A N be the projection map, omitting the coordinates Xi for all 
indices i with = modulo to'. Performing the construction of 2.2 we may 
choose T = T(m, N, a) C A N and V = T(m', N', a') C A N ' such that p maps T 
to T . (The map T -> T is then dominant.) If /': C -> T' is the family of 
curves associated with the triple (to', N',a'), the pull-back p*C" = C'xt'T can 
be identified with the quotient of C/T modulo the action of p, r C p m , where 
r = to/to'. 
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Let V = Y(m',N',a') be the Q-VHS over V associated with the triple 
(to', N', a'). The quotient morphism C — > p*C over T gives a map p*V -> V 
and this induces an isomorphism p*Y' = ©dlm'V^. We refer to the sub-Q- 
VHS p*V as the old part associated with the divisor to'. The direct summand 
yM c V is called the new part; in the decomposition (4.2.1) it is the only 
direct summand that is not contained in an old part. 

(4.4) Lemma. — Let the notation and assumptions be as in 4.3. Let g = 
g(m, N, a) and g' — g(m', N', a') be the respective genera. If Z(m, N, a) C £^ g .c 
is a special subvariety then Z(m',N',a') C &? g ',c * s special, too. 

Proof. Suppose Z(m, N, a) is special. Write J — J(m, N,a) — > T and J' = 
J(m',N', a') — > T' for the respective Jacobians. Then p*J' is an isogeny factor 
of J. Hence, if 9: T — > srf g ' <c is the morphism corresponding to p* J', it follows 
from 1.4 that the Zariski closure of the image of 8 is a special subvariety. But 
9 is just the composition of the projection p: T — > T' , which is dominant, and 
the morphism 4>{m' , N', a'): T' — > £^ g \c associated with the data (m',N',a'). 
Hence the closure of the image of 9 is Z(m' , N',a'). □ 

(4.5) With notation as in 2.5, the C-local system Vc decomposes as Vc = 
©nez/mzVc,(n)- The relation with (4.2.1) is that vj? is the sum of all V c .(„) 
with ord(n) = d. We have Vc,(o) = 0) an( i for n ^ the C-local system V c .(„) 
has rank N(n) — 2, where N(n) is defined as in 4.1. (See also [7], Section 2.) 
With real coefficients, and with /(to) as in (3.2.1), we have a decomposition of 
K-VHS 

v R = v Ri( ± n) ] e¥ R , (?) , 

where the last summand only occurs if to is even. The decomposition is such 
that 

V R ,(±n) = V C ,(„) © V C ,(_„) 

and 

Vr.(^.) ®r C = Vc,(ffi) (for even m) . 

On the summand V K; (±„) the polarization of the VHS induces a (— l)-hermitian 
form f3± n of signature (d n ,d- n ); see also [7], Corollary (2.21) and (2.23). In 
case to is even, the polarization induces a symplectic form (3m. on Y^m.y 
As in 3.3 we choose a base point b 6 T(C), and a symplectic similitude 
u: _ff 1 (J;,,Q) V. Via this similitude, we identify V with the fibre of V 
at b. Correspondingly, we write Vr ; (±„) for the direct summand of Vr that 
under a maps to the fibre at b of ~V^± n y 
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For what follows, it will be convenient to choose the base point b to be a Hodge- 
generic point with respect to the variation V, so from now on wc assume this. 
Define Mon C GL(V) and Hdg C GL(V) to be the algebraic monodromy groups 
of the Q-local system V and the Hodge group of the fibre at b, respectively. 
(Since we have chosen b to be Hodge-generic, Hdg is the generic Hodge group 
of the VHS.) By [3], Theorem 1, the identity component Mon is a semisimple 
normal subgroup of Hdg. 

(4.6) Remark. — The generic Hodge group Hdg is contained in the algebraic 
group H := GLq^j (V) H CSp(V, ^) considered in 3.3. Extending scalars to R 
we have 

(4.6.1) H R =l H U(V R , (±n) ,/3 ±n ) xS P (Vk ( ™ ),/%), 

\±nel(m),2n^0 J 

where the symplectic factor only occurs if to is even. Our formula (3.3.1) follows 
from this together with what was explained in 1.5. 

We have Mon C Hdg C H . The next proposition gives us information about 
the projections of the M-group Mon R onto the various factors in the decompo- 
sition (4.6.1). 

(4.7) Proposition. — (i) Suppose we have ±n G I(m) with 2n ^ such that 
d n ^ 1 and d_ n ^ 1. Then the image o/Mon R in U(Vr r± n ), P±n) is the special 
unitary group SU(Vr ) (±„), /3±„), which is isomorphic to SU(c?„, d- n ). 

(ii) // to is even, the group Mon R projects surjectively to the symplectic group 
Sp(V5 Rj (^), /%); which is isomorphic to Sp 2 , lR with h — dm.. 

Proof. For (i), see Rohdc [22], Theorem 5.1.1. For (ii), assume to is even, and 
let h := dm . There are then 2h+2 indices i for which is even, and without loss 
of generality we may assume these are the indices i G {l,...,2ft + 2}. With the 
above notation, V Ri (m) is the same as V R \ and if wc define Mon [2] C GL(VM) 
as the algebraic monodromy group of the Q-local system V' 2 ' then the image 
of Mon in Sp(V Kj (m), fim.) equals MonJ 2 '' . Further, is just the local system 
obtained from the family of hyperelliptic curves u 2 = (x — ti) ■ ■ ■ (x — ^2/1+2); 
cf. 4.3. By [1], Theorem 1 (see also [2]), the connected algebraic monodromy 
group of this local system is the full symplectic group. □ 

(4.8) Let (to,7V, a) be a triple as in 2.1, with N ^ 4. Consider the following 
condition. 

f a q "i \ There exists a ±n G /(to) with {d n , d- n } = {1, iV — 3}, 

(4. o.l ) 

and for all other ±n' G /(to), cither d n i — or cL„/ = 0. 
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All examples in Table 1, with the exception of Example 2, satisfy this condition. 
It has been proven by Rohde in [22] that these nineteen examples are the only 
examples (up to equivalence) of triples (to, N, a) with TV > 4 that satisfy (4.8.1). 

(4.9) We now prove Theorem 3.6 for m = 2. In this case N is even, g = 
(N — 2)/2, and up to equivalence a = (1, . . . , 1) is the only possibility. Further, 
Z(2, N, a) C st g ,c is the closure of the hyperelliptic locus in si g . The assertion is 
that this is not special for g ^ 3. This is true because for g > 3 the hyperelliptic 
locus is not dense in #/ g , whereas by (ii) of Proposition 4.7 the generic Hodge 
group Hdg is the full symplectic group. 

In what follows we may therefore assume to > 2. 

5. Canonical liftings of ordinary Jacobians in positive characteristic 

(5.1) We need to recall some definitions and results from the paper [8] by 
Dwork and Ogus. We first discuss their theory in a general setting. After that, 
from 5.4 on, we return to the specific families of curves that interest us, and 
we explain how the results of [8] apply to that situation. 

Given an irreducible base scheme T and a smooth projective curve /: C — > T 
we write ujc/t f° r ^c/t- Consider the Hodge bundle E = E(C/T) := f*ooc/T> 
which is a vector bundle on T of rank g, the genus of the fibres. We have a 
Kodaira-Spencer map KS: Sym 2 (E) -> and we define 

X = XiCjT) := Ker(Sym 2 (E) /.(wg/ T )) , 

£ = J2{C/T) := Coker(/*(u;® 2 T ) v Sym 2 (E) v ) , 

where "mult" is the multiplication map. We remark that JS can be thought 
of as the pull-back to T of the normal bundle to the Torelli locus inside s£ g . 
Outside the hyperelliptic locus, mult is surjective and ,J4f is the dual of £1. 

(5.2) Let k be an algebraically closed field of characteristic p > 0. Consider a 
smooth projective curve C/k of genus g such that its Jacobian J is ordinary. 
Let A be the natural principal polarization of J. By Serre-Tate theory (see [9] 
or [10], Chap. 5) we have a canonical lifting of (J, A) to a principally polarized 
abelian variety ( J can , A can ) over the ring of Witt-vectors W{k). In general, for 
g ^ 4 this canonical lifting is not the Jacobian of a curve over W(k). More 
precisely, Dwork and Ogus show that in general not even the canonical lifting 
over W2(k), the Witt-vectors of length 2, is again a Jacobian. 

Following [8] we call the ordinary curve C/k pre-Vv 2- canonical if there exists 
a smooth projective curve Y over W2(k) such that (J can ,A can ) over W2(k) is 
isomorphic, as a principally polarized abelian variety, to the Jacobian of Y. 



16 



Ben Moonen 



The main advantage of working modulo p 2 is that in this case there is a natural 
class flc/k m £{C/k) that vanishes if and only if C/k is pre-WVcanonical; see 
[8], Prop. (2.4). This invariant fic/k cannot be defined in a family of curves (in 
a sense that can be made precise, see [8], § 3), but its Frobenius pullback can, 
as we shall discuss next. 

Let T be a smooth fc-scheme, and let Ft- T — > T be its absolute Frobenius 
cndomorphism. Consider a smooth projective curve f:C^T such that all 
fibres Cf are ordinary. This assumption permits us to view the inverse Cartier 
operator as an ^V-hnear homomorphism 

7: F^E -> E , 

with E = E(C/T) the Hodge bundle. This map 7 is the inverse transpose of 
the Frobenius action on R 1 f*&c- 

Dwork and Ogus define a global section 0c /t of F^^(C/T) such that for any 

t G T(k) the value of (3 C/T at t is F*((i Ct/k ). Sec [8], Prop. 2.7. 

The sheaf F^.JS{C /T) comes equipped with a canonical flat connection 

V: F*£{C/T) -»• F*£>{C/T) ® Q} T/k . 

In particular, still with T smooth over k, iiC t /k is pre-WVcanonical for all t G 
T(k), the section flc/T is zero, so certainly Vfic/T = 0. One of the main points 
of [8], then, is that V/3c/t can be calculated explicitly. The result is easiest to 
state under the additional hypothesis that the fibres Ct are not hyperelliptic. 
We note that this assumption is not always satisfied in the situation we want 
to consider. In Prop. 5.8 we shall give a modified version of the next result 
that also works for hyperelliptic families. 

If the Ct are not hyperelliptic, the multiplication map Sym 2 (E) — > /*(w® 2 ) is 
surjective, so, with notation as introduced in 5.1, Pc/t can be viewed as an 
^ T -linear map F^X -> G T . 

The following result is [8], Prop. 3.2. (In [8] the result is stated under some ver- 
sality assumption on the family C/T but one easily verifies that their Prop. 3.2 
is true without this assumption.) 

(5.3) Proposition (Dwork- Ogus). — With assumptions and notation as above, 
and assuming the curves Ct to be non-hyper elliptic, —'V(/3c/t) : F^J^f — > S!y/ fe 
is equal to the composition 

F*jr c™*> ^Sym 2 (E) S ym 2 (E) ^ ft* . 
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(5.4) Assumptions. In the rest of this section we fix data (m,N,a) as 
in 2.1 with m > 2 and TV ^ 4, and we consider the family of curves /: C — > T 
as constructed in 2.2. We assume that (m,N,a) is not equivalent to one of 
the triples listed in Table 1. We further assume that the closed subvariety 
Z(m 7 N,a) C £& g ,c is a special subvariety; our aim is to derive a contradiction. 
Without loss of generality we may suppose (to, N, a) is minimal with respect 
to the above assumptions, by which we mean that there is no triple (to', N' , a') 
that satisfies our assumptions and for which we have 4 N' < N or N' = N 
and 2 < m' < to. In particular, it follows from Lemma 4.4 that if to' is a 
proper divisor of to and (m',N',a') is obtained from (m,N,a) by reduction 
modulo to', as in 4.3, either N' ^ 3 or (m',N',a') is equivalent to one of the 
twenty triples listed in Table 1. 

(5.5) Lemma. - (i) The subsheaf /*(w®, T )(o) of (i m -invariant sections of 
,f*(w® 2 T ) is a locally free &t -module of rank N — 3. 

(ii) The Kodaira- Spencer morphism KS: Sym 2 (E) — > ti^ZR f ac t° rs over the 
composite map 

Sym 2 (E) ^ Sym 2 (E) (0) ^ /*(< T ) (0 ) 

and the induced map KS( ): /*( w ®/ T )(o) ~^ ^t/r * s a fibrewise injective homo- 
morphism of vector bundles. 

Proof (i) By a result of Chevalley and Weil in [5], for every t G T(C) the 
subspacc of /i m -invariants in H°(Ct, £l® 2 ) has dimension N — 3. 
(ii) The fibre of Sym 2 (E)( ) at a point t is dual to the space H 1 (C t , 6)(o) ; which 
parametrizes the deformations of C t for which the ^i m -action deforms along. 
Since we have a /x m -action on the whole family C/T, the Kodaira-Spencer 
map factors through the projection Sym 2 (E) — > Sym 2 (E)( ). For any family 
of curves it also factors through the multiplication map Sym 2 (E) — > /*(cl)® 2 s ); 
hence KS factors through /*(w® 2 T )(o)- The last assertion of (ii) just says that 
at any t £ T(k) our family is complete in the sense of deformation theory: if 
D — > Spcc(fc[e]) is a first-order deformation of Ct with its /x m -action, D/k[e] 
can be obtained by pull-back from our family. This is clear, as the quotient 
D/fi m is isomorphic to PL e j. □ 

(5.6) Lemma. — There exists a prime number p with p = 1 modulo m and, 
for p a prime of R = Z[l/m, u]/$ m above p, a dense open subset U of T := 
T®r(R/p), such that for any algebraically closed field k of characteristic p and 
any t e U{k) the curve C t is ordinary and the Jacobian J t is pre-Wi- canonical. 

Proof. Let p be any prime number with p = 1 mod to. Note that in this case 
R/p = F p for any prime p C R above p. We write T ® ¥ p for T ® R (R/p). By 
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[4], Prop. 7.4, there is a Zariski open U C T ® F p such that C t is ordinary for 
all t G U(k). On the other hand, by a theorem of Noot [18], [19], in the slightly 
more precise formulation of [13], Thm. 4.2, the assumption that Z — Z(m, N, a) 
is special implies that for p large enough and t e To(k) any ordinary point, the 
canonical lifting of J t gives a W(k)-v&\ued point of Z. In particular, J t is then 
pre-M^-canonical. □ 

(5.7) For most choices of (to, N, a) the general member in our family of curves 
C — > T is not hypcrclliptic. In this case we may, and will, choose p and U in 
Lemma 5.6 such that the curves C t with t £ U(k) are non-hyperelliptic. 

It may happen, however, that Z(m, N, a) is fully contained in the hyperelliptic 
locus, in which case we call (to, N, a) a hyperelliptic triple. This occurs, for 
instance, if to = 2m' is even and a is of the form (ai, ai, to', . . . , to'), or if 
N = 4 and a = (1, 1, to — 1, to — 1). In such a case we will just choose p and U 
as in Lemma 5.6. 

The following variant of Prop. 5.3 is the essential tool in our proof of Theo- 
rem 3.6. Let p and U be chosen as above, and let Cjj — > U be the restriction 
of C/T to U. With notation as in 2.5 and 5.1 we have eigenspace decompo- 
sitions J2{Cjj/U) — ®nei/mi =^(n)i likewise for other [/^ m ]-modules such as 
X(CujU) and Eu = E(Cu/U). Accordingly we can write P Cu /u = J2 n /V)> 
with /3(„) a section of F^JS^y 

(5.8) Proposition. — With assumptions and notation as in 5.4 and 5.7, the 
composite map 

(5.8.1) 

F* v J(f (0) ^ J$gym a (%) ( o) Sym 2 (%) ( o) /.(«^)(o) 

is zero. 

Proof. If (m, iV, a) is non-hyperelliptic this is immediate from Prop. 5.3 to- 
gether with (ii) of Lemma 5.5. So from now on we may assume that Z(m, N, a) 
is contained in the hyperelliptic locus. Let t £ Aut(Cu /U) be the hyperelliptic 
involution. We remark that i may or may not be contained in the subgroup 
Hm C Aut(C;y/[/); by inspection of the hyperelliptic examples mentioned in 5.7 
we see that both cases occur. 

Let oj = Vl 1 Cu ^ u . In the hyperelliptic case the multiplication map Sym 2 (E^) — > 
/*(w® 2 ) is no longer surjective. However, if we denote the invariants under the 
action of t by a subscript "even" , 

mult (cvcn) : Sym 2 (Ec/) (ovcn) ->• /*(w 02 )( C vcn) 

is again surjective; see for instance [21], Lemmas 2.12 and 2.13. The assump- 
tion that Z(m, N, a) is fully contained in the hyperelliptic locus implies that 
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/»(w® 2 )( ) C /*(w® 2 )( cvon ). Hence also mult( ) is surjective, and we may view 
/3( ) as an ^[/-linear map F^Jf^ — > ffjj. 

Our assumptions imply that /3( ), hence also V/?( )) is zero, so all that remains 
to be checked is that for the 0-component the analogue of Prop. 5.3 holds. 
For this we can follow the proof of [8], Prop. 3.2: The proof that loc. cit. dia- 
gram (3.2.9) is commutative does not use the assumption that the curves are 
non-hypcrclliptic and therefore goes through in the hyperelliptic case. More- 
over, in the notation of that proof, we can choose the lifted family Y/T (where 
Y/T is our Cu/U) such that it has an action of p, m . Taking O-components 
in diagram (3.2.9) then gives that — V/3( ) equals the composition of (5.8.1) 
and KS(o), and by (ii) of Lemma 5.5 we conclude that the map (5.8.1) is 
zero. □ 

(5.9) Lemma. — Write q = (p — l)/m. Let n G Z/mZ, and let A = A n (t) G 
GLd n {&u) be the matrix of the Cartier operator 7 -1 : JE[/,(n) ~~ ► Fu^u,(n) with 
regard to the frames 

u n fi, • • • , w»i„-i , respectively io nfi <g> 1, . . . , w n ,d n -i ® 1 • 

Then the matrix coefficient A p , a , for p, a G {0, . . . , d n — 1}, equals 



JN 
N ! 



(5.9.1) (-If- J2 ( q - [ - nai]m )...( q - [ - naN]m ).t?...t 
where S = (d n — a)(p — 1) + (p — a). 

Proof. This is the dual version of [4], Lemma 5.1, part (i). □ 

We remark that the result of Bouw in [4] is valid in a more general context. She 
uses this to prove the existence of cyclic covers whose p-rank reaches a natural 
upper bound imposed by the local monodromy data; see [4], Thm. 6.1. 



6. Proof of the main result: four branch points 

In this section we prove Theorem 3.6 in the case = 4. Let us first try to 
explain the idea of the argument, by way of guide for the reader. 
Assume we have a triple (m, N, a) that is not one of the twenty examples in 
Table 1 but such that Z(m,N,a) is special. We want to show that this is 
impossible. We have the family of curves C — > T over a ring R of finite type 
over Z; we then consider the reduction modulo p, with p chosen as in 5.7. The 
nontrivial information we have, exploiting the assumption that Z(m, N, a) is 
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special, is that this gives a family of ordinary Jacobians in characteristic p such 
that the canonical liftings over W2(k) are again Jacobians. In Prop. 5.8 we 
have translated this, based on the theory in [8], into the vanishing of a certain 
abstractly defined map. Lemma 5.9 enables us to calculate this map explicitly. 
To apply this, we need a differential form that gives us an element in the source 
of the map (5.8.1); we write down such a form in 6.2 below. The fact that the 
image of this element is zero then gives us a polynomial identity (6.2.1). By 
some combinatorial arguments we then show that this identity cannot hold if 
(m, N, a) is not equivalent to one of the twenty special triples in Table 1. 

(6.1) We retain the assumptions made in 5.4. In addition we assume N = 4. 
Let 

S>{m, a) := {±n £ J (to) \ d n = d_ n = l} . 

(Here I(m) is as defined in (3.2.1) and d n is given by (3.2.2). Though not 
indicated in the notation, d n depends on a.) 

If ±n e @(m,a), the equalities d n = d- n = 1 imply that nai ^ modulo m 
for all i £ {1, . . . , 4}. Hence, 

4 

(6.1.1) [—nai] m =m—[naj\ m for alH, and ^ [— naj} m = 2m , 

i=l 

where we recall that [b] m denotes the representative of the class (b mod m) in 
{0,l,...,m-l}. 

We first observe that dim S([i m ) > 1. Indeed, if dimS(/z m ) = 1 then (4.8.1) 
holds, and by the results of [22], Chapter 6, (m,N,a) is one of the triples in 
Table 1, which contradicts our assumptions. 

For n £ Z/mZ we have d n + d- n < 2, and if m is even then dm < 1. The 
inequality dim S(fj, m ) > 1 is therefore equivalent to the fact that #^(m, a) > 2. 
Choose two distinct pairs ±n and ±n' in 3>{m, a). Note that if m is even, we 
may have n = —n or n' = —n'. 

(6.2) Choose p and U as in Lemma 5.6, where we may further assume that 
the curves Ct for t £ U (k) are either all non-hypcrclliptic or all hyperelliptic. 
We keep the notation introduced in the previous section; in particular we recall 
that X(C v jU) = ffi n£ z/ mZ J( n ). 

It follows from (6.1.1) and the definition of the forms oj n ^ in (2.6.1) that 

V ■= Un,0 ® U- n fi - UJ n < fi <g> U-n',0 

is a section of J^o)- F° r v € {±n, ±n'} the matrix A v = A v (t) of Lemma 5.9 is 
a polynomial in ¥ p [ti, . . . ,t^\. As the Jacobians of the curves C t for t £ U{k) 
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are ordinary, the Cartier operator 7" 1 in Lemma 5.9 is an isomorphism of 
vector bundles, so A v {t) is invertible as a section of Because uj n .o ■ ^- n ,o = 
oJn',0 ■ W- n 'fi is a non-zero section of f*(uj® 2 ), it follows from Prop. 5.8 that 

(6.2.1) A n ■ A- n = A n , ■ A_ n , 

as polynomials. 

Define B v := A„\ tl =o, the polynomial obtained from A„ by substituting t\ = 0. 

Explicitly, 

(6.2.2) 

b v = ( -iy 1 . y ( q - [ - l/a2]m ) ■■■( q - [ - l/aiU ) ■ t^t{ 4 ■ 

Fix an index t e {2, 3, 4}, and write {1, 2, 3, 4} = {1, t} II {k, A}. Let v n (i) be 
the largest integer v such that B n is divisible by t v L . From (6.2.2) and (6.1.1) 
we find 

v„(l) = max{0, (p - 1) - q ■ [-na K ] m - q ■ [-na A ] m } 
= max{0, q ■ [na K ] rn + q ■ [na\] m - (p - 1)} , 

and similarly for V- n {i). 

Next let w± n (b) be the largest integer w such that B n ■ B_ n is divisible by tf. 
We find 

w± n (t) = max{0,q • [naj m + q ■ [na x } m - (p - 1)} 

+ max{0, q ■ [-na K ] m + q ■ [-na\] m - (p - 1)} , 

which by (6.1.1) we can rewrite as 

w± n {b) = max{0, q ■ [na K ] m + q ■ [na\] m - (p - 1)} 
(6.2.3) + max{0,<7 • [nai] m + q ■ [na t ] m - (p - 1)} 

= q ■ max{[nai] m + [noJ m , [na K ] m + [na x } m } - (p- 1) . 



(6.3) Lemma. - Consider two 4-tuples (b\, 62, &3, ^4) and {bi,b' 2 ,b' 3 ,b' 4 ) in 
{1, . . . , m — l} 4 with Y^ii=i bi — 2m = 5Z i=1 ^- Suppose that for all partitions 
{1,2,3,4} = {1,l}U{k,X} we have {h + b L ,b K + b x } = {K + K,b' K + b' x } as 
sets. Then there is an even permutation o G A4 of order 2 such that either 
b'i = b a (i) for all i orb' i =m — b a (^ for all i. 

Proof. Straightforward checking of all 8 — 2 3 possible combinations (two for 
each partition). □ 
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(6.4) The relation (6.2.1) implies that for any partition {1,2,3,4} = {l,i} H 
{k, A} we have w± n (i) = w± n >(i). Since Yh=i \ na i]™ = 2m = Yh=i \ n ' a i]m, it 
follows from (6.2.3) that 

{[nai] m + [na L ] + [nax] m } = {[n'ai] m + [n'a L ] m , [n'a K ] m + [n' a x ] m } 

as sets. By the lemma it follows, possibly after replacing n by — n, which 
we may do, that there is an even permutation a £ A4 of order 2 such that 
[ji'fli]m = bi<V(i)]m f° r a U As the pairs ±n and ±n' are distinct, a ^= 1. 
Possibly after a permutation of the indices we may therefore assume that 

. . [n'ai] m = [na 2 ] m , [n'a 3 ] m = [na 4 ] m , 

(6.4.1) 

[na 2 ] m = [na\] m , [n a 4 ] m = [na 3 ] m . 
Since gcd(m, 01, . . . , 04) = 1 this implies that gcd(m, n) = gcd(m, n'). 

(6.5) Let r = gcd(n, m). Let m' = m/r, and consider the triple (m' , N' ,a') 
obtained from (m,N,a) by reduction modulo ml. Then we have a bijection 
3>(m,a) 2>(m',a r ) by (±n) i-> (±2). Hence #@(m',a') > 2. In particular, 
iV' = 4, and (m', AT', a') is not equivalent to one of the triples listed in Table 1. 
By our minimality assumption on (m,N,a), see 5.4, it therefore follows that 
r = 1. Hence we may replace the 4-tuple a by n'a = (n'ai, . . . , n'ai), which 
means we may set n' = 1 in the above. In this case, (6.4.1) tells us that n 2 = 1 
in (Z/mZ), and the 4-tuple a has the form a = (a\, nai, 0,3, 110,3) with n/il 
and 

(6.5.1) (n + l)(ai +a 3 ) = 2m. 

From these relations we want to deduce that there is a pair ±1/ £ 3>{m, a) with 
gcd(m, v) ^ 1, thereby obtaining a contradiction. Write m = 2 k ■ M with M 
odd. 

First assume n ^ — 1 modulo M. For an odd prime number I and an exponent 
e ^ 1, the congruence n 2 = 1 mod £ e only has n = ±1 as solutions. Hence 
there is an odd prime divisor t of m such that n ^ — 1 mod £ But then (6.5.1) 
gives a\ + 03 =0 mod £, and since gcd(m, a\, . . . , 04) = 1 we further have 
ai ^ mod £ and 03 ^ mod £. Taking ^ := raji we find that ±v £ ^(m, a); 
contradiction. 

The only remaining possibility is that n = — 1 modulo M, and therefore n ^ 
— 1 mod 2 fe . In particular, k ^ 2. In this case (6.5.1) implies ai +a 3 = mod 4, 
in which case ±m/4 £ ^(m, a); contradiction. This completes the proof of 
Thm. 3.6 in the case N = 4. 
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7. Proof of the main result: five or more branch points 

We now turn to the case N > 5. The idea behind the proof is the same 
as in the previous section, but there are some additional difficulties. In 7.2 
and 7.3 we prove two technical results. Once we have these, we can again write 
down elements in the source of the map (5.8.1). Similar to what we did in 
Section 6, this gives us polynomial identities (7.4.1), and with some elementary 
combinatorial arguments we can then conclude the proof. 

(7.1) Assumptions. — We retain the assumptions made in 5.4. In addition we 
now assume N > 5. Again we assume (to, N, a) is minimal. Because we have 
already proven Thm. 3.6 for N = 4, it is again true that if to' is a proper divisor 
of m and (to', N' , a') is obtained from (to, TV, a) by reduction modulo m! , either 
N' ^ 3 or (to', N', a') is equivalent to one of the twenty triples listed in Table 1. 

(7.2) Lemma. - With assumptions as in 7.1, there exists an index n E 
(Z/mZ)* such that {d n , cL„} ^ {0, N - 2}. 

With the terminology introduced in 4.3, the lemma says that the new part of 
the local system V (as in Section 4) is not isotrivial. 

Proof. We assume {d n ,d- n } = {0, N - 2} for all n 6 (Z/mZ)*, and we 
seek to derive a contradiction. It will be convenient to consider the function 

S a : Z/mZ -> Z^ given by S a (n) = Y^Li ( = ^ L )- Note that for 71 ^ the d « 
defined in (3.2.2) equals —1 + 5 a (n). For n € (Z/mZ)* we have S a (n) ^ 1 and 
5 a (n) + S a (— n) = N. Our assumption is equivalent to the condition that 

(7.2.1) {5 a {n),6 a {-n)} = {l,N -1} for all n e (Z/mZ)*. 

Possibly after replacing a — (ai,...,ajv) by —a, we may assume S a (l) = 1, 
which means that 

(7.2.2) MmH \-[a N ] m = m. 

We divide the proof into a couple of steps. 

Step 1. Our first goal is to show that m > 60. For a given to, (7.2.2) leaves only 
finitely many possible triples (to, N, a), up to equivalence. With the help of a 
small computer program we check that for to ^ 60 the only two possibilities 
with N ^ 5 and {S a (n), 6 a (-n)} = {1,7V - 1} for all n e (Z/mZ)* are given 
(up to equivalence) by 



(7.2.3) 



to = 6, TV = 5, a = (1,1,1,1,2); 
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(7.2.4) m = 6, N = 6, a = (1, 1, 1, 1, 1, 1) . 

So it remains to show that for these triples, Z(m 7 N,a) is not special. 

In case (7.2.3), we have g = 7 and the family of Jacobians J — > T decomposes, 

up to isogeny, as a product of three factors: J ^ Yi x Y 2 x Y 3 , with 

Y\ a pull-back of the Legendre family of elliptic curves, which is Example l 
in Table 1; this is the old factor corresponding to the divisor 2 of m, 
Y 2 a pull-back of Example 6; this is the old factor corresponding to the 
divisor 3 of m, 

— Ys the new part. 

(The new part Y3 is in fact an isotrivial family of 3-dimensional abelian varieties 
of CM type, but we will not need this.) Let b 6 T(C) be a Hodge-generic point. 
The fibre Yi.t, is an elliptic curve with endomorphism ring Z; its Hodge group 
is isomorphic with SL2. The fibre Y 2) b is an abelian threefold that is easily seen 
to be simple. (E.g., use the fact that Example 6 is special, together with [23], 
Thm. 5.) By [16], Prop. 3.8 the Hodge group of Y\j> x Y" 2 ,6 is the product of the 
two Hodge groups. Hence the smallest special subvariety S C si-j that contains 
Z(m, N, a) has dimension at least 1 + 2 = 3 (see 1.5), and since N — 3 = 2 < 3 
this means that Z(m,N,a) is not special. 

In case (7.2.4), we have g = 10 and the family of Jacobians J — > T decomposes, 
up to isogeny, as a product of three factors: J ~ Y\ xY 2 x Y~ 3 , with 

Y\ a pull-back of Example 2; this is the old factor corresponding to the 

divisor 2 of to, 

Y 2 a pull-back of Example 10; this is the old factor corresponding to the 
divisor 3 of to, 
Y3 the new part. 

(In this case the new part I3 is an isotrivial family of 4-dimensional abelian 
varieties of CM type; again we will not need this.) For a Hodge-generic point 
b e T(C) the fibre Yi^ is an abelian surface with endomorphism ring Z; its 
Hodge group Hi = Hdg(Yi.b) is isomorphic with Sp 4 . The fibre Y 2 ^ is an 
abelian fourfold. By Thm. 1.1 of [26] we have End(Y2.fc) = ^[£3], and the 
tangent multiplicities for the factor Y 2 (=Example 10) are given by d{\ mo d 3) = 
3 and di 2 mod 3 ) = 1. By [15], 7.4, it follows that for the generic Hodge group 
H 2 = Hdg(Y2.fc) we have i?2,R — U(l,3). If H = Hdg(Jft) is the generic 
Hodge group in our family of Jacobians, H projects surjectively to Hi and H 2 . 
It follows that among the simple factors of H^ d both a factor PSp 4 and a 
factor PSU(1, 3) occur; this implies that the smallest special subvariety S C stf-j 
that contains Z(m,N,a) has dimension at least 3 + 3 = 6 (again see 1.5), and 
since N — 3 = 3 < 6 this means that Z(m, N, a) is not special. 
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Step 2. The function S a has the property that 5 a {n\ + n 2 ) < S a (ni) + S a (n 2 ). 
In particular, if m, n 2 and n\ + n 2 are all in (Z/mZ)* then by (7.2.1) we have 
the implication 

(7.2.5) 5 a (m) = 5 a (n 2 ) = 1 =► <5 a (m + n 2 ) = 1 . 

Step <?. Next we show that to is not divisible by 2, 3 or 5. Let p £ {2,3,5}, 
and suppose to is divisible by p. Let to' = m/p and let (m', N', a') be obtained 
from (m,N,a) by reduction modulo m'. The new triple cannot be one of the 
examples listed in Table 1, for this would imply (by inspection of the table) 
that TO = p-m'^5-12 = 60, which we have excluded. Hence our minimality 
assumption on the triple (to, N, a) implies that N' ^ 3. 

Since N ^ 5, there are at least two indices i such that aj = modulo to'. For 
p = 2 this contradicts (7.2.2). Hence 2 £ (Z/mZ)* and by repeated application 
of (7.2.5), starting from S a (l) = 1, we find that <5 a (2 fc ) = 1 for all k. 
For p = 3 the only possibility left, in view of (7.2.2), is that there are precisely 
two indices i with [ai] m = to/3 and that [a,j\ m < to/3 for all remaining indices. 
This contradicts the fact that 5 a (2) — 1. Hence 3 does not divide to. 
Finally take p = 5. The fact that S a (2) — 1 implies that there is a unique 
index i with [a,j\ m > to/2 and that [oj] m < to/2 for all remaining indices. 
Since N ^ b and N' ^ 3 this leaves (possibly after a permutation of the 
indices) two possibilities: either 

Mm = 3to/5 , [a 2 ] m = to/5 , [<Xj] m < to/5 for all i > 2, 

or 

to/2 < [ai] m < 3to/5 , [a 2 ] m = [a 3 ] m = to/5 , [aj] m < to/5 for all i > 3. 

In both cases we get a contradiction with the fact that 5 a {&) = 1. 

Step 4- We now show that to has at most four distinct prime divisors. To 
see this, let p be any prime divisor of m, let to' = m/p and let (m',N',a') be 
obtained from (to, TV, a) by reduction modulo to'. If the new triple is one of the 
examples listed in Table 1, it can only be Example 17, for otherwise inspection 
of the table gives that to' (and hence to) is divisible by 2, 3, or 5, which we 
have excluded. Example 17 has N' — 4 branch points, so this fact, together 
with our minimality assumption on the triple (to, N, a), implies that N' ^ 4. 
The previous argument shows that for any prime divisor p of to, there are at 
most 4 indices i such that a, is not divisible by m/p. On the other hand, 
a-i ^ mod to, so for a given index i there can be at most one prime divisor p 
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of m such that dj is divisible by raj p. As A ^ 5 it follows that m has at most 
four distinct prime divisors. 

Step 5; conclusion of the proof. Combining the conclusion of Step 3 with (7.2.5), 
we find that for all p E {1,2,3,4,5} we have (p mod m) £ (Z/mZ)* and 
6 a (p mod m) = 1. 

Let ^ be an integer such that gcd(m, v) — 1 and <5 (V mod m) = 1. Let p be 
the smallest positive integer such that gcd(m, v + p) = 1. The fact that m 
has at most four distinct prime divisors, all greater than 5, implies that p ^ 5. 
But then gcd(,o, m) = 1 and from (7.2.5) we get that 8 a (v + p mod m) = 1. 
Repeating this, we find that 5 a (n) = 1 for all n e (Z/mZ)*, which contradicts 
the fact that S a (n) + 5 a (—n) = A > 3. This completes the proof of the 
lemma. □ 

(7.3) Our next goal is to show there exist two distinct pairs ±n and ±n' 
in J(m) with 

(7.3.1) d- n = gL„< = 1 , and d n = d n > = A — 3 . 

(In particular, n ^ and n' ^ 0.) For this we work over C and we consider 
the generic Mumford-Tate group M of the family C/T. As in 1.5, let M^ d = 
Qi x • • • x Q r be the decomposition of M^ d as a product of simple factors. 
The assumption that Z(m 7 N,a) is special means that N — 3 = Ym=i d{Qi), 
with notation as in 1.5. Further, this assumption implies that the connected 
algebraic monodromy group Mon° of the family is a normal subgroup of M; in 
particular, Mon[^' ad = riieif Qi f° r som c subset K C {1, . . . , r}. 
By Lemma 7.2 there exists an index n in (Z/mZ)* with {d n , d- n } ^ {0, A — 
2}. By Prop. 4.7 it follows that one of the simple factors Qi, say Qi, is a 
PSU(c?„, cL„), which gives d(Qi) = d n d- n . Because n £ (Z/mZ)* we have 
d n +d- n = A — 2; hence d n d- n ^ A — 3 with equality of and only if {d ni d- n } = 
{1, A — 3}. Possibly after replacing n by — n we therefore have d n = A — 3 
and d- n = 1. Further, the relation A — 3 = Ya=i d{Q%) implies that all other 
simple factors Q2,...,Q r are anisotropic (i.e., compact). 

Our assumptions imply that there is another index pair ±n' £ I(m) with 
d n > ^ and cL n ' ^ 0, for if this is not the case then (m, A, a) satisfies (4.8.1), 
which implies it is one of the examples of Table 1. If m — 2m' is even and 
n' = m' then by Prop. 4.7 one of the factors Qi is a PSp 2/l with h — d n > > 0. 
As A — 3 ^ 2, this factor cannot be the factor Qi = PSU(1, A — 3), and since 
PSp 2 /j is not anisotropic we arrive at a contradiction. So n' ^ —nf, and, again 
by Prop. 4.7, one of the Qi is a non-compact unitary factor PSU(d„', d- n >). 
This factor must be the factor Qi; hence {d n >, d- n >} = {1, A — 3}, which gives 
what we want. 
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(7.4) Consider two distinct pairs ±n and ±n' in I(m) for which (7.3.1) holds. 
Let T 6 G~Ln-3(&u) be the matrix of jf n y. F^Kui n \ — > Et/,(„) with regard to 
the frames given by the forms w„ i(7 . It is the inverse of the matrix A — A n of 
Lemma 5.9. Similarly, let V be the matrix of r )i n ')i an d let A' — A n i be its 
inverse. Further, let c and d be the sections of (the inverses of the lxl 
matrices A_„ and A_„/ of Lemma 5.9) such that 7(w_ ni o <8> 1) = c • cj_„, an d 
7(w_„',o ® 1) = c- w_„ %0 - 

For a G {0, 1, . . . , N - 4}, let r\ a G r(Z7, J^ )) be given by 

rycr := cj_„, ® w„ ;Cr - L0- n >,o ® w„/ )(T . 

As cj_„.o • w„ iJ( , = u>-n',o • <*V i(9 as sections of /»(w® 2 ), the image of ^ un- 
der Sym 2 (7) equals 

JV-4 

Because the sections w- nj o'W n . p for p G {0, . . . , N — 4} are linearly independent, 
it follows from Prop. 5.8 that c • T p><7 - c' ■ T' pa . for all a, p G {0, 1, . . . , N - 4}, 
which can be rewritten as 

(7.4.1) c- 1 -A Pi<r =J<- 1 -A' Pi<r . 

Choose two distinct indices k, X in {1, 2, . . . , N}, and write {l,...,iV} = 
{k, A} II /. For v e {0, 1}, define v n (v) to be the largest integer v such that 
A„ lV \t K =o is divisible by t v x . Similarly, let w_„ be the largest integer v such that 
c _1 \t K =o is divisible by t v x . Using the explicit formulas for c _1 and the matrix A 
from Lemma 5.9, we find 

v- n = maxjfj, (p-l)-q-^2 [« a »]™} , 

iei 

v n (0) = niaxjfj, (N - 3)(p - 1) - q • ^ [-naj] m | , 

t>n(l) = 0. 

(Recall that q = (p — l)/m.) 

Next we define w n (v) = v_„ + u n (i^) to be the largest integer u> such that 
(c -1 • j4^ ;I/ ) |t„=o is divisible by t\". Similar to (6.1.1), it follows from the fact 
that d- n = 1 and d n = N — 3 that {—nai] m — m — [nai] m for all i and 
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Ei=l i na i\m = 



2m. Using these relations we obtain 



w(0) 



max|(p -l)-q-^2 [n<k]m, (p - 1) - 1 ■ ^2 ^ na ^ m 




w(l) 




maxjo, (p - 1) - g • ^ [na;] m j = 9 • max{0, [na K ] m + [na\] 

10 



rn 




We can do the same calculations for the pair ±n'; let us call w'(0) and w'(l) 
the resulting values. From (7.4.1) we get the relations w(0) — w'(0) and 
w(l) = w'(l). It follows that for all choices of n and A we have [na K ] m + 
[««A]m = [n'a K ) m + [n'a\] m . This readily implies that [nai] m — [n'ai] m for all 
i G {1, . . . , N}. As gcd(m, ai, . . . , ajv) = 1 it follows that n = n', which contra- 
dicts our assumption that the pairs ±n and ±n' are distinct. This completes 
the proof of Theorem 3.6. □ 
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